Exploring Advanced Techniques for Training an Artificial Neural Network on a Complex, Multimodal Game Scenario

I conduct a comparative study of different techniques (standard backprop, complementary reinforcement backprop, NEAT) for training a recurrent neural net on a difficult problem. I contribute an additional layer of strategy by offering the net access to expert algorithms, which can provide domain knowledge about the task at hand. My hypothesis was that learning how to make use of this expert knowledge would prove an easier task than learning the problem from scratch. Results support this hypothesis, but indicate that experts also introduce further complication. Artificial neural network (ANNs), inspired by the biological circuitry of the human brain, are one of the oldest and most versatile algorithms for machine learning. Academics and industry professionals have successfully applied them to a wide variety of learning tasks, spanning the gamut from facial recognition to e-mail spam filtering and more. By and large, the basic structure of neural nets has remained the same over many years.1 However, effective training of a neural net can require a fair amount of wrangling, and different approaches have proven most successful in different contexts. I am interested in pushing the limits of neural nets by placing them in a context where they have traditionally done poorly, and then exploring new options for training. The context I chose for this study is a puzzle game called Mastermind. It presents two primary challenges as a learning task: 1. Good moves require a memory of all previous moves that have been made 2. Optimal strategy changes as the game progresses In theory, neural nets with “recurrence” – that is, connections which relay information backwards – are capable of factoring previous decisions into future ones. However, the depth and persistence of this kind of memory is a subject of ongoing research, and neural nets tend to perform best when short-term memory is unimportant. Recent advances in “deep learning” are the most notable exception. Neural nets can also suffer from the issue of “catastrophic forgetting” – that is, even their long-term memory can fail, provided significant changes in environmental conditions (like the shifting game strategies of Mastermind). Some research has tried to address this problem by performing clustering analysis, and forking off different local experts to learn each cluster.2 The defining concept for this study was to apply a localexpert-based model to a recurrent neural network. However, rather than evolving experts from scratch, I wanted to determine whether the network could learn to consult preestablished experts in its domain. These experts were hardcoded with a variety of standard strategies for playing Mastermind. None of them could play an entire game in an optimal fashion, but each possessed some knowledge that might be useful during particular game phases. It is worth noting that Mastermind was an especially convenient candidate for this kind of supervised learning, because of how easy it is to verify, for any given move, which expert would have offered the best advice. This study has potential implications both for training ANNs on multimodal learning tasks in general, and for integrating pre-existing domain knowledge into the learning process. Ideally, it would be possible to use these techniques to take experts trained in CBIM (or some equivalent algorithm) and apply them to new learning contexts. My work also offers promise for making optimal use of redundant control systems – for example, programming an aircraft to automatically know when to trust its pilot and when to trust its co-pilot instead. Multiple experts are only better than one provided that they are each allowed to play to their respective strengths. A well-trained ANN might be able to classify exactly what those strengths are and when to rely on them.